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TO THE EDUCATIONAL RESOURCES 

Overview -j information center (eric) 

The Decision Dinner - a parable about the weighted average decision strategy 
by Bruce Landon and Nancy 

The novelty of the proposed decision process can be illustrated by a parable about a small group of decision makers and 
relating it back to the online learning product selection situation. First, assume the task of selecting a restaurant for a board 
of directors’ first dinner in a new city. Also imagine that this dinner is pretty important, perhaps not quite as important as the 
Nixon dinner in China, but a dinner of consequence, nonetheless. The first steps in the process would be to scan the 
available resources to see which restaurants (like product providers) are available and then to proceed to get menus (lists of 
the features available in each provider’s product). The menu information could be augmented with reviews of the items on 
the menu but this would still not be enough. What the decision making group needs to know is what kind of items the 
members of the board actually like to have for dinner, i.e. which menu items are important to them. So the group decides to 
collect some more information by sending out a stakeholder survey to find out what each of the board members enjoys for 
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dinner. 



To construct this survey, the decision making group had to make a short list of restaurants (product providers) and from their 
menus make a short list of the most promising items (important features) because the board members would have never 
responded to a survey that was as long as all of the menus from every restaurant. The group sent out its survey and 
analyzed the results to calculate the importance weighting for each of the dinner menu items. Knowing the consequences of 
making a mistake on selecting a restaurant for both the decision makers and the board of directors, they elect to use the 
most rational decision making strategy known: the weighted averaging strategy. 



To be able to use this strategy, the decision makers will have to find out how suitable the menu items actually are. So they 
have dinner at each of the restaurants to taste and judge the quality for themselves. Not only do they have to carefully taste 
the suitably of the menu items (features) but they must rate each one on a consistent scale so that the scores can be used in 
the weighted average strategy. They decide to all use a scale where 0 means awful and 10 means wonderful on suitability 
rating forms used when they go to the restaurants. Then night after night, dinner after dinner, they evaluate the suitability of 
each of the short listed restaurants on all of the important menu items. Finally using the board members’ weights for each 
menu item and the decision makers’ suitability scores for each menu item at each restaurant, they calculate the winner from 
the data collected. 



The winning restaurant is the one that has the highest average score when the menu item weights are multiplied by the menu 
item suitability scores. The decision makers make the reservations with the winning restaurant and the board of directors 
proceeds to have a great dinner. The board of directors was impressed by the rationality of the group of decision makers but 
in truth they were more impressed by the dinner after which they voted to give bonuses to the decision makers and all live 
happily ever after. 



Selecting an online eductional delivery application is analogous to selecting the restaurant in the parable above. The 
members of the board of directors are analogous to the stakeholders who will be impacted by the selected product i.e. the 
students, faculty, staff, and administrators. If the consequences of making an error and selecting the wrong product were not 
important there would be no need to be so rational as to use the weighted average decision strategy and the group of 
decision makers could use a less rational process. When the consequences are important such as in the assigning of course 
grades then a very similar rational process is most often used called weighted averaging grading. 



This chapter will focus on faculty use of a decision-making process for complex situations. The analysis part of the process 
describes and compares course management software focusing on: technical specifications, instructional design values, 
tools and features, ease of use, and standards compliance. The extensive comparisons provide faculty with side-by-side 
product feature descriptions. The decision-making focus of the chapter and of the companion site is on supporting a detailed 
rational decision process for selecting course management software. This decision process uses a grading style model 
familiar to educators that compensates for the cognitive illusions and limitations of decision makers to achieve a transparent 
decision process for selecting the "best" application for each local situation. The current site obviously meets needs for 
decision-making support as evidenced by 16k visitors a month to the site. By the fall of 2002, there will be similar decision- 
making supports available which will focus on student services, teaching technologies and online education policies. 

^ Introduction 

Faculty make decisions every day and most of them are informal choices about what looks good on the menu for lunch. 

O Sometimes, faculty make quite formal and rational decisions about which student gets an A and which student gets an F 
^ ^'®'ng a strategy of weighted averaging. The way in which decisions are made often reflects the importance of the decision 
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and the situation as well as the decision maker. The research in psychology over the last few decades has illuminated some 
of the strengths and weaknesses in the way that people make decisions with the important finding that in many situations 
people are less than rational decision makers. When a person is overloaded with information or under time pressure then 
their decision-making can be quite compromised and they seem to use “rules of thumb” to make a decision. The case of 
selecting an online educational delivery system is examined from this perspective in this chapter. First the difficulties facing 
the decision maker are addressed and then a remedy is demonstrated to show how decision-making can be assisted to 
prevent the decision process from being compromised by these “rules of thumb” using a web tool for comparative analysis. 

Why choices are difficult 

Too many products 

The situation is reminiscent of the early automobile industry when there were over a hundred automakers. Like the early 
auto industry many other competitors will not survive in the long run. The current front-runners: WebCT and BlackBoard 
started out as small almost personal projects a few years ago and are now multi-million dollar enterprises that span the global 
education marketplace. There may well be others out the current products that will evolve into the major brands of tomorrow 
as the market matures. The point of this analogy is that selecting a learning management system is much more complicated 
than selecting a new car, in part, because there are so many competing models. There are over 100 products in this 
marketplace that range from relatively simple limited function applications like WebBoard which just provides threaded 
discussions and chat to complex enterprise wide learning systems that link to backend databases and administrative data 
systems used by colleges and universities. Previously in 1997, I had attempted to characterize these products as 
components or suites by this categorization turned out not to be useful because there were so many variations from product 
to product he became difficult to classify new products. Also, the products in this marketplace evolve quite rapidly with a new 
model or version almost every year, like automobiles. So to deal with this rapidly changing market it seems better to 
characterize products in terms of the features that they provide. 

Too many product features 

The feature and tools are organized in terms of the group that they are most likely to be interested in them: Learners, Learner 
Supporters and Technical Administrators. These major groupings are further divided into subtypes of features/tools and 
finally into the individual features of the product. This schema has grown over the years and the language has changed as 
well so that some of the terms have been replaces in common usage by other terms as in the case of Newsgroup being 
supplanted by the term discussion or forum. A glossary of terms provides some help for those who are unfamiliar with this 
area (http://wwww.c2t2.ca/landonline/Qlossarv.htmn . but the fundamental problem is that there too many features/tool about 
too many products to keep track of easily (products * features = 3474 facts) especially to keep track of in your head. The 
most frequent problem in making good decisions in this kind of situation is that of forgetting or omitting some important part of 
the problem when making the decision. Often we are overly optimistic about how big a decision problem we can do “in out 
head.” 

Too little working memory in decision-makers 

Nobel laureate, Herbert Simon (1979) characterized the human decision making capacity as having “Bounded Rationality” 
(also see Gigerenzer & Selten, 2001). When people face decisions they operate within the bounds or capacities of the 
human mind. They are limited by fallible perceptions, wandering attention, faulty memories, and fluctuating information- 
processing abilities. People don’t optimize, but instead resort to simplifying rules of thumb in order to proceed with their 
decision-making. These limitations have been investigated for decades with the repeated finding that working memory 
limitations play a major role in limiting the ability to make rational decisions. 

The notion of limited memory was articulated well by George Miller (1953) as the magical number seven plus or minus two 
“things” that you hold in your mind at once while working on a problem (presumably due to a phonological loop process, 
Baddeley 1986, cited in Anderson 2000). This limited working memory is a profound handicap for a rational decision maker 
because when one works with ideas in the head then one is moving them around sort of like a juggler. When there are too 
many “things” then some of them will get dropped or set aside from consideration in the decision-making. Stated in another 
way, decision-making becomes more difficult as the number of things to consider increases and quickly reaches a point of 
overloading working memory. 

When facing difficult decisions, people encounter overload and then they resort to ways of coping that involve cognitive 
shortcuts. When overloaded, they are more susceptible to cognitive illusions (Kahneman and Tversky, 1996). Cognitive 
illusions are similar to perceptual illusions in that they stem from the fact that in both types of illusions the person can focus 
only on a small part of the situation at one time. In the case of perceptual illusions this is because the fovea of the eye that 
sees fine detail encodes only a small segment of the spatial panorama at each fixation. In the case of cognitive illusions this 
is largely because the conscious working memory can hold only a few “things” and the rest must be inferred or extrapolated 
in ways that are sometimes biased. In both types of illusions the person is not necessarily aware of any “illusion” at all and 
may believe that they are “seeing everything correctly.” 



Too many Cognitive iiiusions in Decision-Making 

The discrepancies associated with cognitive Illusions may be seen a little more clearly in contrast to an idealized rational 
decision process. In the idealized process, one starts by selecting relevant features/criteria and then assigning an 
importance weighting to each of those features. Next one evaluates each choice option on each of those important features 
and assigns a suitability score. Finally one makes the rational decision to choose the option with the highest weighted 
average suitability score (the average of the weights for a feature multiplied by the feature’s suitability score). Choosing the 
option with the best score is the rational decision (Keeney and Raiffa, 1976 as cited in Roe, Busemeyer and Townsend, 
2001). With only a couple of optional choices and a few important features it is likely the limitations of working memory would 
be overwhelmed even if one were able to do the mental multiplication part. When one is overwhelmed by the size and 
complexity of the decision situation then one is susceptible to a host of cognitive illusions. A few of the major ones are: 
Availability Heuristic, Representativeness Heuristic, Hindsight Bias, Gambler’s Fallacy, the Framing Effect, Similarity Effect, 
Decoy Effects, and Overconfidence Effect. When the decision situation is simple and small, on the other hand, most people 
can and do make good judgments unaffected by these cognitive illusions. 

The Availability Heuristic investigated by Tversky and Kahneman (1973) is a powerful cognitive distortion. Essentially, it is 
captured by the by the saying, “out of sight - out of mind.” It turns out that the believing the truth of a piece of information is 
related to the ease of recall. If something does not easily come to mind then the inference is that it must not be true (Begg, 
Amour and Kerr, 1985). Unfortunately, the ease of recall is affected by circumstances that have nothing to do with the truth 
of the information such as the vividness of the information and the number of times it has been repeated. The power of the 
Availability Heuristic is not lost on those who repeatedly advertise on radio and television. 

Another powerful cognitive illusion is the Representativeness Heuristic (Kahneman and Tversky, 1973). The illusion is that If 
something looks like one of those then it must be one of those in spite of relevant base rate information to the contrary. 
Common versions of this are "you can tell a book by its cover” and “if it looks good then it works well.” These rules of thumb 
leave out the base rate information such as about how likely it is for a random book to be a good read. The cognitive illusion 
is to be unable to process additional information about the likelihood of an event given the overwhelming impact of how it 
looks on the surface. 

The Hindsight Bias is the tendency to believe, after the fact, that one would have foreseen a particular event (Fischhoff, 1982 
as sited in Galotti, 1999). This has also been referred to as the “l-knew-it-all-along-phenomenon” that inflates self-esteem 
and feelings of overconfidence in one’s predictive abilities. The notion is that hindsight is always 20-20, but in reality, many 
events like the weather are still uncertain predictions before the fact. 

There is an entire industry built on the cognitive illusion of the Gambler’s Fallacy. This is the popular illusion of believing that 
in a game of chance that the more you loose the greater your chance of winning (Tversky and Kahneman, 1971). The truth 
is that the odds of winning stay the same. The illusion is belief that the law of large numbers applies in the situation of small 
numbers and that there would necessarily be some balance in the situation of a specific gambler. This shows up as the 
gambler’s belief that in a fair game of chance after you have lost for a while it becomes more and more likely that it is your 
turn to win. Other forms of this can occur in committees who believe that after a string of poor decisions it is their turn to get 
lucky in picking a good vendor or whatever has been the focus of their decision-making. 

The Framing Effect refers to the impact of the context on the decision. People tend to avoid risks that are described In terms 
of benefits, but tend to take the same risks that are described in terms of loss (Tversky and Kahneman, 1981). This Is 
reminiscent of the rule of thumb: win - stay, lose - shift. The illusion is that the value of losses is greater than the value of 
gains when mathematically they are equal. This misinterpretation of the importance of risk can bias the importance weighting 
of features and thus bias the decision process. 

Similarity Effects (Tversky, 1972) and Decoy Effects (Roe et al, 2001) are both products of confusion. When a new option is 
added to a choice set that is similar to one of the existing options the original option is less likely to be chosen because this 
effect sort of divides the mind share like a similar product can divide the market share. The Decoy Effect is a special case of 
the Similarity Effect where a range decoy that is more extreme than the original choice set dramatically reduces the choice 
likelihood of nearby options (Roe, 1999). In effect, the extreme decoy sabotages the nearby options by altering their context. 

Decision Deadlines can also influence the role of Cognitive Illusions. When deadlines are quite short the decision maker 
tends to rely more on fewer pieces of information. In effect the working memory is further limited by time pressure. Roe’s 
(1999) analysis suggests that the option chosen under time pressure is simply the one that is the current focus of attention 
when time runs out. With longer decision deadlines the more subtle similarity effects exert more influence. Longer decision 
timelines in the context of face-to-face group decision making tend to be associated with more extreme decisions due to 
attitude polarization (Moscovici and Zavalloni, 1969). It is rarely the case that truth wins in the group decision-making 
situation (White, 1993 as cited in Myers and Spencer 2001). 

Overconfidence is a paradoxical Cognitive Illusion in that it is the illusion that one has no cognitive illusions and that one’s 
decisions are without bias. Kahneman and Tversky (1996) have found a replicated tendency for people to be more confident 



than is warranted by the evidence and to overestimate the accuracy of their beliefs. This overconfidence extends to 
eyewitnesses testimony where Loftus (1979) has found that the witness’s confidence in their testimony to be unrelated to 
accuracy of their testimony. This overconfidence has the side effect of enhancing personal self-esteem and thereby 
contributing to resistance to being influenced otherwise i.e. stubbornness. 

In summary, there are many mental obstacles for those who would be rational decision makers. Many of these Cognitive 
Illusions stem from the limitations of working memory. These are the same limitations that cause people to not be able to 
remember long strings numbers like international telephone numbers. Because large and complex decisions situations are 
beyond the capacity of working memory, people resort to rule of thumb strategies to do the best they can. In such 
overwhelming situations Cognitive Illusions bias judgments and compromise decision-making. 

Making the decision situation smaller and simpler can circumvent these obstacles. Dividing up a complex decision into 
several small and simpler decisions keeps the load on working memory within normal capacity and then decision makers can 
be quite rational in making good decisions. 

Donald Norman (1999) has made the case that human-computer interactions can and should be designed to take account of 
normal human fallible perceptions, wandering attention, faulty memories, and fluctuating information-processing abilities. 

With good design of decision-making processes that keep the in-the-head part small and simple, a system can be optimized 
to enable quality decision-making. The normative weighted averaging strategy can be easily divided up into small and simple 
component parts and then later recombined (outside-of-the-head) into a rational decision process that leads to the "best 
option." There is a powerful decision-making synergy in combining the focused judgmental prowess of people with the 
memory and computational prowess of computers. 

How people normally Make Decisions: The 5 Basic Strategies 

There are five basic decisions strategies (Payne & Bettman, 2001) for choosing among options (products, services etc.) 
based on features or aspects of the options. All of these five strategies are compromised by the reality that the set of options 
is limited to those known to the decision-maker and the “best” option may be simply missed in the whole process. While the 
names of these strategies on may be unfamiliar is quite likely that everyone has used all of the following strategies at one 
time or another: . 

Lexicographic strategy (aka one reason, pick the best) 

Elimination by aspect strategy (aka pick the last) 

Satisficing strategy (aka Bounded Rationality Model) 

Equal weight strategy (aka scoring strategy) 

Weighted averaging strategy (aka weighted adding strategy, grading model) 

The Lexicographic strategy involves two steps: selecting the most important feature and then picking the product from all of 
the options that is best on that one feature. There is no requirement for the features to be expressed as numbers. If there is 
a tie then the decision maker simply repeats the process on the second most important feature. This non-compensatory 
strategy does not allow for some features of an option to make up for other less adequate features. The only involvement of a 
second feature is in the case of a tie that is broken by considering the next most important feature and selecting the option 
that is best on that feature. 

The elimination by aspect strategy is a simple screening strategy involving two steps: setting the requirements for each of the 
features and then eliminating options one at a time which do not meet any one of the requirements. The decision maker 
examines the features of each option one by one and the option is eliminated as soon as a feature is found that does not 
meet the requirements. This popular strategy does not need for the features requirements be expressed as numbers and is 
easy-to-use. The elimination by aspect is a non-compensatory strategy so that once an option is rejected on any feature that 
option is eliminated from further consideration. This strategy does not necessarily result in a single best option since more 
than one option may pass on all of the requirements. Including any features that are irrelevant or biased in favor of one of the 
options compromises this process. - 

The Satisficing strategy (Simon, 1955 cited in Gigerenzer and Selten, 2001) is more psychologically developed model of 
sequential decision-making that includes setting requirement cutoff levels for each feature. An option is examined until it fails 
to meet one of the cutoff levels then it is rejected and the next option is considered. The first option that passes on all of the 
features is selected. If none of the options pass all of the cutoff requirements, then the requirements are reduced to new 
lower cutoff levels and the process is repeated. The satisficing strategy is non-compensatory and effected by the order in 
which the options are presented as well as the aspiration level of the initial cutoff requirements. 

The Equal weight strategy is comprehensive and considers all of the features on all of the options. The suitability values of 
one (passed) or zero (failed) are assigned to each to each feature on each option. The total suitability score for each option 
is the simple sum of the featue suitability values for that option (all features are weighted equally). The option with the 
highest sum of the values is selected. This strategy is directly analogous to multiple choice test scoring where correct 
glowers are awarded one, wrong answers are awarded zero, and the highest sum score identifies (selects) the best student 



ort the test. This compensatory numeric strategy enables some features assigned one's to compensate for other features 
that were assigned zero's. This strategy is more useiil when there are many features because there is less risk of a tie for 
best score. 

The Weighted averaging strategy is a more complex version of the simple sum score model where each feature is also 
assigned an importance weighing. This idea implies that the decision maker is willing to make trade-offs to arrive at the 
selected option. The process involves five steps: (1) setting the importance weights for each of the features, (2) for each 
option then determining the numerical suitability value for every feature, (3) multiplying the feature suitability value times the 
feature weight, (4) then summing these weighted features subscores into an option total score, and (5) finally selecting the 
option with the highest average score. The averaging of the weighted option total scores simply returns the scores to the 
scaling metric of the feature suitability value units (which are often on a rating scale such as 1=poor to 9=excellent). This 
strategy is directly analogous to the grading model where different parts of the grade are weighted differentially (or 
subsections of an exam are weighted differentially) to arrive at the final mark. The weighted averaging strategy is 
compensatory in two ways: high scores on one feature compensate for low scores on another feature and a lower importance 
weight on one feature is compensated for by a higher importance weight on another feature. This strategy is less likely to 
result in ties between options than is the equal weight strategy when using a relatively small number of features. 

Furthermore, the weighted averaging strategy is considered the normatively rational decision process because it uses all of 
the information available in a consistent manner to arrive at a selection. This model is similar to the multiple linear regression 
model used in statistical analyses and suffers from some of the same shortcomings, most notably it is compromised when 
irrelevant features are used and when unreliable suitability values are used. 

Decision makers often make use of combinations of the above strategies. In combined strategies the decision is made in 
phases. The initial phase is often to screen out some of the potential options in order to reduce the complexity of the 
situation to as short list of options. This phase is followed by a more thorough consideration phase to select the best option 
from a short-list of candidate options. The primary reason that everyone does not use the weighted averaging strategy all the 
time for all decisions is because it would be too mentally difficult and too time consuming. Other strategies are heuristic 
attempts to make good decisions with less effort and in less time and sometimes they ^eem to work well enough. 

idealized decision-making process for comparison 

Selecting a online educational delivery application is a high-stakes gamble that can impact the future prosperity of an 
educational institution. The weighted averaging strategy is ideal decision-making process because it is considered to be the 
only normatively rational decision process. In its most rational form every possible product option would be considered on 
every possible feature - nothing would be left out. The process involves the five steps: (1) setting the importance weights for 
each of the features, (2) determining the numerical suitability value for every feature of every product option, (3) multiplying 
the feature suitability value times the feature weight, (4) then summing these weighted features subscores into a product 
option total score, and (5) finally selecting the product option with the highest average score from among all of the possible 
product options. In this ideal there are three major difficulties: discovering every possible product option in a dynamic 
expanding market, accurately determining the importance weightings of each of the features, and accurately determining the 
suitability of each feature on every product. The practical difficulty is simply not having sufficient time and resources to carry 
out the process. The recommended process is below intended to approximate this ideal strategy within the practical 
limitations of time and resources. 

Tailoring decision processes to Individual Situations by involving the stakeholders 

The ultimate success of a decision process in addition to making a good decision is that others, the stakeholders, cooperate 
with the decision makers and carry out the decision and implement the chosen alternative. Tetlock (1983 as cited in 
Gigerenzer 2001) has studied the social aspects of rationality and proposed three social goals that are important for 
maintaining this cooperation. The first social goal is that decisions are made in a transparent way that is understandable and 
predictable. The second social goal is fairness where the decisions are made in a way that does not violate the expectations 
of people. The third social goal Is where decisions are made in an accountable way that can be justified and defended in 
public. Decisions, which meet all three of these social goals, are more likely to pave the way for the acceptance by the 
stakeholders of the eventual decision to select a particular product over the competition. 

One direct way of accomplishing these goals is to involve the stakeholders in the overall decision process. Including several 
of the stakeholders on the decision-making or decision-recommending committee often does this. The common difficulty with 
this approach is that there are many times more stakeholders than there are members on the committee. By dividing up the 
decision-making tasks it is possible to meaningfully involve more stakeholders and still keep the decision-making committee 
to a workable size. There are a couple of obvious groups of stakeholders: the student learners and the faculty who will be 
using the selected system to deliver online courses or use it to deliver the online portion of blended regular courses. In 
many institutions there will also be other stakeholders who will be interested in some involvement in the decision process. 
Other obvious stakeholders are the systems and computing personnel who would be doing the technical administration of the 
Learning Management System if it is not hosted from the vendor’s site. Also, there may be a myriad of learner support 
services from the registrar to instructional design personnel to the library and disabled student services personnel who all 
‘y*"' a stake in the decision process. Finally, depending on the level of involvement the institutional Board of Directors may 

ERIC 



6 



wfell have political role to making their priorities felt in such an important institutional decision. One recommended approach 
is to involve some of the stakeholders in deciding the importance of specific product features and also involving the more 
technical expert stakeholders in judging the suitability a specific feature in a particular product. 

When the initial importance weightings of the product features for the comparative analysis site were developed there were 
many differences of opinion among the experts involved in the peer review committee. These differences were worked out 
and consensus was achieved by using a couple of face-to-face meetings followed a nominal group process similar to the 
Delphi process (Delbecq, Van deVen and Gustafson 1975). After people met and had an opportunity to talk about what they 
thought was important and why, they were then asked to provide importance ratings for the specific features. These ratings 
were simply averaged to provide a mathematical consensus without resorting to additional face-to-face meetings. The 
resulting averages were also sent back to the peer review committee by email with requests for further comments if members 
had strong feelings that there was an error anywhere. These requests produced some new ratings that were again averaged 
and circulated again for comment and revision. It took only a couple of rounds of this nominal group process to produce a 
consensus on the feature weights. This consensus was in part because the process was transparent, fair, and accountable 
as recommended by Tetlock (1983, cited in Gigerenzer 2001). 

Focusing on what is important 

Colleges and Universities are hugely different as are institutions of 3,000 students and institutions of 30,000 students. 
Historically, much of educational market has been a geographical patchwork of niche markets with some exceptions. In each 
institutional situation of there are some considerations that are more important than others. One way to approach the problem 
of too many products and too many features is to limit the consideration to only those things that are locally important. 
Determining what features are needed in an institutional Learning Management System is often a political process in each 
institution that can involve many people. This process is informed by the institutional history of what has worked well and 
what has not, as well as expectations about what will be needed in the future. In practice, different institutions and faculty 
within those institutions use products in different ways and in more than a few cases some features of a product are never 
used. This situation results from the newness of the market and the rapid pace of technological change in recent times. In 
this situation the wiser approach would seem to be to focus on what is needed rather than what is offered in the market. The 
idea of a needs analysis is not a new idea, but it can be particularly advantageous in selecting a Learning Management 
System by informing the process of which product features fit into the institution’s needs and which ones are less needed or 
merely duplicate existing institutional systems. 

Each institution is unique and that uniqueness can be augmented or diminished by the choice of an institutional Learning 
Management System. Sometimes institutional mission and goal statements can provide a guide to which directions to 
pursue and sometimes these policy documents play no real role in the decision making of the institution. What is at issue in 
the selection of a Learning Management System is the potential commitment to a product that is as far-reaching as the 
institutional decision about which word processor application to support. Once the decision is made it will not be easy or 
inexpensive to undo that decision. By basing the decision around the unique needs of the specific institution there is a better 
chance that decision made will further the success of the institution rather than the opposite. 

Thinking in terms of the audiences of users 

One way to reduce the complexity of the decision process is to divide the product features into groupings based on the group 
of people who are more likely to be involved with those features. In this instance one can think about three different groups 
of people: Learners, those who support the learners more or less directly and those who are involved in the technical 
administration of the software installation. These three groups do not exhaust the possibilities but provide a starting 
viewpoint. 

The learners or students are usually the largest group of users. It is useful to consider the Learning Management System 
from their perspective since any problems or short comings are going to effect so many people and because many 
institutions profess a “learner focus” as part of their mission. However learners are not a homogeneous group and the needs 
of “first time users” may well be very different from those of returning successful students. 

The Learner Support group includes faculty, teaching assistants, instructional designers, various administrators and others 
who’s job it is to help learners to be successful. The titles and descriptions of these roles vary dramatically from situation to 
situation 

« 

The audience of technical administrators consists of those folks who operate the server as well as those who administer the 
authorization or security policies out of sight. In many situations is this audience also includes additional administrators 
involved in setting up course shells on the server and arranging for access to registration lists. In general, the technical 
administrators operate behind the scenes and are able to access parts of the software system that are not available to 
instructors or course developers. 
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Thinking in terms of product features 

Within each of the three audiences the features of the products can be further grouped into clusters of features and this can 
reduce the complexity further. On the Online Educational Delivery Applications site fhttD://www.c2t2.ca/landonline/ ) the 
following conceptual groupings have been used to provide targeted views for the learners, learner support, and technical 
administrator audiences respectively: 

Learner Tools (for learner audience) 

Web Browsing 
Asynchronous Sharing 
Synchronous Sharing 
Student tools 

Support Tools (for learner support audience) 

Course 

Lesson 

Data 

Resource 
Administration 
Help desk 

Techlnfo (for technical administrator audience) 

Server Platform 
Client Platform 
Pricing 

Limitations of package 
Extra Considerations 

Within each of these conceptual categories there would be a small number of specific application tools or features. 

Altogether there were 62 such specific features that could be a part of an individual application. The complete listing of 
application features inside of the above conceptual structure attempts to organize the otherwise overwhelming list of features: 
(adapted from http://www.c2t2. ca/landonllne/option2. asp) 



Check Features to include 



Learner_Tools 


Support_Tools 


Techlnfo 


Web_Browsing 


Course 


Server_Platform 


□ 


Accessibility 


^ Course planning 


□ 


RAM 


o 


Bookmarks 


^ Course managing 


□ 


Disk Space 


□ 


Multimedia 


^ Course customizing 


□ 


WindowsNT 40 Server 


□ 


Security 


^ Course monitoring 


□ 


Apple Server 


Asynchronous_Sharing Lesson 


□ 


Unix Server 


□ 




^ Instructional designing 


E-mail 


Client_Piatform 



^ BBS file exchange 

^ Newsgroups 

Synchronous_Sharing 

^ Chat 

Voice Chat 
Whiteboard 
AoDlication sharing 
Virtual space 
Group browsing 
Teleconferencing 



□ 

□ 



Presenting information 



Minimum Level 



Testing 



□ 

□ 

o 

□ 

□ 

□ 

ERIC 



Data 

□ 

□ 



Markina on-line 
Managing records 



□ 

^ Target Level 

Pricing 

^ Start‘UP Cost 



^ Analyzing and tracking 

Resource 

□ 



□ 

□ 



On-aoina Cost 



□ 

□ 



Curriculum Managing 
Building knowledge 
Team Building 



Technical Support 

Limitations of package 

□ 



□ 

□ 



IMS Compliance 
Number of courses 
Number of students 
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Videoconferencing 

Student^tools 

□ 



□ 

□ 

□ 

□ 



Self-assessing 
Progress tracking 
Searching 
Motivation building 
Study skill building 



^ Building motivation 

Administration 

□ 



Installation 

Authorization 

Registering 

On-line fees handling 
Server security 
Resource monitoring 
Remote access 



n 
□ 

□ 

□ 

□ 

□ 

^ Crash recovery 

Heip_desk 

n 



n 



student support 
Instructor support 



^ Number of connections 
^ Number of instructors 

^ Other Limitations 

Extra.Considerations 

^ Options 

^ Exit Considerations 



No product had all of the possible features and no institutional situation needed all of the features either. The Online 
Educational Delivery Applications site was intended to help institutions find products with the features that they need and for 
vendors of products to be discoverable on the basis of the product functionality in the context of a rapidly developing 
marketplace. The list of product features also turns out to be a moving target over time as new technologies are incorporated 
into the evolving applications. 

Product Reviews and Rapid Product Change 

The situation of finding appropriate application candidates is made substantially more difficult by the fact that the products 
themselves are changing. Most of the products will release a new version one or more times per year and in many situations 
this “revision” can change the product substantially, especially when the newness comes from a company merger or 
acquisition. Published product reviews and comparisons quickly become dated in spite of the best of intentions. This 
dynamism among the vendors is heightened further by technological changes such as the advent of database back-ends on 
courseware systems that has occurred during the last several months. 

In the context of the field of rapidly moving products it is very difficult be base decisions on what is available, but it is still quite 
feasible to base decisions on what is needed in the individual situation. The institutional context of online learning is 
changing more slowly than the marketplace and It can be argued that a needs analysis is more valuable than a market 
analysis in such a situation. 

Screening phase of decision-making based on requirements of Individual Situations 

The elimination by aspect strategy is a screening strategy Involving two steps: setting the requirements for each of the 
features and then eliminating options one at a time which do not meet any of the requirements. To be able to decide which 
features are important the conventional approach is to conduct some sort of needs analysis of what is required in the 
institutional setting. This analysis often reveals the uniqueness of the needs of the future users of the online system. At this 
stage it can be beneficial to involve the stakeholders: students, faculty, student support personnel, and the technical 
administrators to gain later acceptance of the eventual decision as well as to become aware of needed features that might 
have been otherwise overlooked. Politically it can be beneficial to even involve representatives from the governing board of 
the institution when this decision will represent a new pedagogical direction or is expected to involve significant long-term 
resource commitment. 

Situational Requirements 

Individual situation are both nominally different and substantially different. Sometimes the differences are so large that our 
tendency to attend to differences makes it seem like some institutions are as different as day and night. While there are a 
host of differences between institutions there are also some similarities so that examining an example of decision-making can 
be instructive about similarities in the processes. As the online educational delivery applications have begun to move from 
the one-size-fits-all mold of the industrial era, they are more adaptable to individual situations but still the applications have 
different strengths and weaknesses. The ideal of making decisions based on individual local institutional situations is to 
exploit the synergy of matching the strengths of a product to the specific needs of the specific situation. The uniqueness of 
local situation can be described in many ways but by using the features of the products as the focus for this description the 
•^♦^ntial synergy is easier to find and exploit. This involves, for example, translating a local need for better two-way 



communication between instructors and students into the importance of the e-mail feature. Likewise the need for orienting 
and assisting instructors in how to use an online educational delivery system can be translated into the importance of an 
instructor help desk feature. When this feature importance is expressed as an importance weighting then this information 
can be utilized in the weighted average decision strategy which combines information about many such attributes into a 
overall suitability score as Is described in more detail in the example below. 

The deciding of which features are more important versus which features are less important based on an analysis of the local 
situation goes a long way toward reducing the risk that the chosen application will be accepted by the stakeholders and will 
benefit the institution. Including irrelevant features at this stage will bias the process in favor of those applications which have 
those irrelevant features so this deciding which features are required is a fundamental part of the overall decision process. 
This is a process where the stakeholders, as described above, can help out by being involved in determining which specific 
features are important enough to be “needs” rather than “wants.” The elimination by aspect strategy does not require that the 
feature requirements be numeric so simply identifying which features are really needed is enough. There are 62 feature 
categories on the Online Educational Delivery Applications site and there are undoubtedly some additional ones are critical in 
some situations. 

The outcome of the deciding which features are important is a Shortlist of important product features. Any product that does 
not include all of those features would not meet the needs of the institution and should be screened out of the competitive 
options for the final selection. This initial product screening can effectively be done by the elimination by aspect decision- 
making strategy. The decision maker examines the features of each option one by one and the option is eliminated as soon 
as a needed feature is missing or completely inadequate. 

Screening tools on the web 

Once the set of needed features has be determined then the Online Educational Delivery Applications site can be used to 
identify which applications provide those features so the number of potential options can be reduced to just those that are 
likely candidates. By using a page like the above table the decision maker can check the checkbox next to the required 
features and then click submit and the returning page will include a list of applications which support those features. Using 
that same page the candidates can be further investigated in side-by-side comparisons of feature descriptions. This side by 
side comparison may help in further reducing the list of applications to only the appropriate options for the local decision 
situation. 



Making a Short List of Application Options 

Using the web tool described above greatly speeds up this elimination by aspect process for those applications that are 
reviewed on the site. The elimination by aspect is a non-compensatory strategy so that once an option is rejected on any 
feature that option is eliminated from further consideration. This strategy does not necessarily result in a single best option 
since more than one option may pass on all of the requirements. The usual outcome of the elimination by aspect strategy is 
to result in a short list of possibly acceptable options that become the narrowed focus for a more discriminating decision 
strategy. It may be prudent for comparison purposes to include in the short list any applications that are used by consortia 
where the local institution is a member or in the case of community colleges the application that is used by local universities 
where the college has special relationships. 

Evaluating product suitability as part of the weighted averaging strategy 

Evaluating products (or externally hosted product options) can be a very large task. The greater the number of products and 
the greater the number of important features multiplies quickly into a potentially overwhelming product research task. This 
task can be partially shifted to the vendors by structuring a competition among the likely candidates where they are invited to 
bring the information to the decision makers rather than the decision makers seeking out the vendors and trying to find out 
the most current product information. The market has matured enough so that most vendors are capable of responding to a 
request for proposal (RFP) to supply the desired product functionality. Many institutions already have RFP procedures but 
may never have used them for a situation as complex as selecting an online educational delivery application. 



Inviting the shortlist of vendors to competitive presentationsiproposals (RFP model) 

The RFP model has been described for web software acquisition recently ( 
httP.7/www.technoloQvnews.net/rfp/infotech rfp.doc) as a necessary part of the decision process. 

The RFP model offers several advantages over less formal product selection processes including: all vendors get the same 
information and have a fair opportunity to compete while the decision makers get a proposal document that can be used to 
assess both the vendor interest and competence. Local procedures may vary but in general a useful RFP should include the 
following sections (according to the info-tech research group at http://www.technoloavnews.net/rfpV 



1) Executive Summary 

2) Reasons for the online educational delivery application so vendors can understand local situation 

3) Explanation of the vendor selection process (the elimination by aspect strategy as above) 

4) Explanation of the proposal evaluation process (the weighted average scoring as above) 
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5) Description of the decision-making team 

6) Description of in-house resources that will also be used in the online educational delivery 

7) Background Information about the institution and the competition 

8) Time line for the vendor’s deliverables 

9) Penalties for missing deadlines 

10) The Specifications including short, medium and long-term specification lists and descriptions 

11) General terms and conditions for the institutional agreements 

12) Special terms and conditions such as the number of references required, legal requirements etc. 

The RFP would be sent to the short list of possible vendors. The vendors will likely have questions as they prepare their 
proposals so it is useful to have a designated contact person who can supply consistent clarifying information to all of the 
vendors’ requests for information. When the proposals are received there is often another round of elimination by aspect 
strategy to determine which proposals are in the competitive range. The US Department of Defense 
fhttD://web.deskbook.osd.mil/reflib/mfarsuDS/072ua/009/072ua009doc.htm1 further advises that a proposal may be 
considered outside of the competitive range if (1) it does not address the essential requirements, (2) has a substantial 
technical drawback the would essentially require a new proposal to fix, or (3) the proposal contains major deficiencies, 
omissions, or out-of-line costs. They also advise that these proposals are part of a negotiation process and at the end of 
negotiations, the competitive vendors should be provided with one additional final opportunity to submit a revision known as 
the “best and final offer.” 

The RFP method is a time-consuming and involving process both for the institution and for the potential vendors that can be 
used to produce a fair vendor competition where the best candidate is selected and the social goals of rational decision- 
making are accomplished. The Online Educational Delivery Application site provides web tool support for the Weighted 
Averaging Strategy calculation part of the of decision-making. This decision engine tool is little more than an automated score 
sheet that combines the feature importance and the feature suitability scores into an appropriate total product score. The tool 
does however enable the decision-makers to use the most rational strategy of decision-making, which would be impossible to 
do in one’s head. The following example illustrates the decision engine web tool using two vendors (WebCT and 
BlackBoard) with three features (accessibility for persons with disabilities, student help, and instructor help). 

Quantifying feature importance in individuai situations 

For the purposes of this example let us assume that our institution is part of the Consortium that is offering us the opportunity 
to use either WebCT or BlackBoard for our existing system. Further lesson soon that there are only three important issues: 
meeting the 508 accessibility regulations so that are federal funding is not jeopardize, meeting the help desk requirements of 
our students, and meeting the help desk requirements of our faculty. We could use the default importance weightings from 
the site that are 1 .0 for accessibility, 1 .2 for student support, and 0.8 for instructor support but it would be better to tailor the 
importance weighting to our institutional setting. So for this example let us assume that we did some campus wide 
information meetings and formed an advisory committee of interested stakeholders. 

Potential of involving stakeholders in setting weights 

Then with the help of our stakeholders we put together a little three-item telephone survey of a random sample students and 
a random sample of instructors. The results of our little survey were importance ratings of 9 for the accessibility item, 7 for 
the student support item, and 6 for the instructor support item. Those numbers can be just entered into the web Decision 
Table form because the form to product the importance weights normalizes the numbers before they are used. 

Judging product feature suitabiiity to iocai situation 

Different persons can make the suitability judgments than were involved in the setting of the importance weights. In this 
example we could conceivably involve the members of our advisory committee to nominate from among themselves a task 
group who have experience with our existing system and are willing to spend a week working with WebCT and a week 
working with BlackBoard to specifically evaluate the accessibility, the student help desk, and the instructor help desk. Further 
our task group could also get others to help with their evaluation as they chose with the only caveat that at least two persons 
with disabilities be involved in checking out the accessibility aspects of the two products. 

If we had allowed more time it might have been possible conduct a request for proposals and have representatives from the 
two vendors provide information and trial facilities to the task group to make their hands-on experience come to them. After a 
week the task group meets with the advisory committee and presents their considered judgments, which are averaged to 
make them more reliable. Their judgment of the suitability of the accessibility for WebCT is 7, for BlackBoard is 8, and for 
the status quo is 2 on a scale of 1 to 9. Their judgment of the suitability of student support for WebCT is 8, for BlackBoard is 
8, and for the status quo is 4. Lastly, their judgment of the suitability of instructor support for WebCT is 8, for BlackBoard is 
6, and for the status quo is 3. Now this is where the doing it in your head is overwhelming with normalizing three numbers to 
an average weight of 1.0, then multiplying nine pairs of numbers and then averaging three columns to arrive at the winning 
Weighted Average Score of 7.59 for WebCT followed by runner up BlackBoard with a Weighted Average Score of 7.45 which 
is well ahead of the status quo at 2.9 socore. 

O 



Demonstration model of the Comparative Analysis Decision Table with Three Options 

(Insert table similar to the one below adapted from http://www.c2t2.ca/landonline/option4.asp) 
Instructions: 

To compare applications A and B just enter your ratings for each criteria into 
the boxes in the column and then press Score. Be sure to enter a rating or a 0 
(for skipping a criteria) for each of the criteria so that the score can be computed. 



Application Evaluation Engine Table for Options: A, B, Status Quo 


Importance 

Weighting 




A 


B 


Stat 




Evaluation Criteria Name 


Webd 

1 


Blacic 

1 


us 










Quo 














9 


Accessibllitv 


7 


8 


2 




7 


Student suoDort 


8 


8 


4 




6 


Instructor suoDort 


8 


6 


3 














ScoreButton 


Weighted Average Scores * 


7.59 


7.45 


2.9 





* NaN - Not a Number error results from a empty rating box in the column Help 



Make current values, the default values 



so you can save your work 



Rechecking by doing Sensitivity Analyses 

One of the benefits of using a computerized process is easy to have the machine perform calculations again. The results In 
the decision table can be recalculated after tweaking the numbers to see if it makes any difference in which one Is the winner. 
The adjusting and recalculating can be done on the basis of the know variability of the numbers Involved to get a better Idea 
of the stability of the solution. In the present example lets say we were curious to see if WebCT would still be the winner If 
the average instructor support rating were one unit lower so we can try a rating of 7 In place of the original 8 and then score 
the whole table again. In this case the change of a rating from a 8 to a 7 would in fact change the winner to BlackBoard 
because the Weighted Average Score for WebCT would drop to 7.32 which is lower than BlackBoard. The point of doing a 
bit of rechecking is that it can become clear as in this case that the products are so close that the small changes In the 
judgments would change the outcome. 

The other rechecking that can be done Is to examine if the outcome would change if the importance weights had been a little 
different. In the present example If the Importance weight for accessibility were changed to 90 from 9 to examine a 10 fold 
increase in the importance of accessibility then BlackBoard would be the winner with a weighted average score of 7.88 
compared to WebCT with a score of 7.13. Notice in both of these instances of rechecking the status quo never wins so one 
reasonable conclusion from this Is that our institution should prepare for a change from the status quo as at least that much is 
quite clearly an outcome of this decision process. 



Suininary and checklist 



The main reason for getting so involved in the decision of which learning management system is the best for an Institution to 
select is that this is a very important decision. Normally when people make a decision or even groups of people make a 
decision they do not make a rational decision. Only when the judgments that people make are quite small and simple can 
they avoid cognitive illusions and biases. Fortunately this kind of product selection decision can be divided up into many 
smaller decisions that can then be recombined into a weighted averaging decision-making strategy that approximates true 
rationality. The trade-off is suggested to first use an elimination by aspect strategy to arrive at a short list of competitive 
products worth considering and then do a more detailed weighted averaging strategy to find out which one has the best 
score. After that the decision table can be further investigated by doing sensitivity analyses to find out how stable the 
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selection of the winner turned out to be. Using the web tools on the site developed for this decision-making can facilitate this 
process: 

http://www.c2t2.ca/landonline 

The online educational deliver application decision checklist: 

1) First decide what features are important in the local situation 

2) Screen options out based on feature criteria using web tools 

3) Making a shortlist of product and features required 

4) Optional Request For Proposals (RFP) from vendors on the shortlist 

5) Judging the importance of features and assigning feature weights 

6) Evaluating the product features for each of the shortlist candidates 

7) Using the Comparative Analysis Decision Engine Tool to pick a winner 

8) Rechecking your work with sensitivity analyses 

9) Communicating how the decision was made to stakeholders 




/ 
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